A Dissertation for the Degree of Doctor Scientiarum An Information Theoretic Approach to Machine Learning

In this thesis, theory and applications of machine learning systems based on information theoretic criteria as performance measures are studied. A new clustering algorithm based on maximizing the Cauchy-Schwarz (CS) divergence measure between probability density functions (pdfs) is proposed. The CS divergence is estimated non-parametrically using the Parzen window technique for density estimation. The problem domain is transformed from discrete 0/1 cluster membership values to continuous membership values. A constrained gradient descent maximization algorithm is implemented. The gradients are stochastically approximated to reduce computational complexity, making the algorithm more practical. Parzen window annealing is incorporated into the algorithm to help avoid convergence to a local maximum. The clustering results obtained on synthetic and real data are encouraging. The Parzen window-based estimator for the CS divergence is shown to have a dual expression as a measure of the cosine of the angle between cluster mean vectors in a feature space determined by the eigenspectrum of a Mercer kernel matrix. A spectral clustering algorithm is derived and implemented in feature spaces defined by the spectrum of the affinity matrix and the Laplacian matrix, respectively. Using tools from statistics for Parzen window-size selection, the new spectral algorithm operates in a fully automatic mode with respect to the width of the Mercer kernel. A connection to the graph cut is also provided. The performance of the new algorithm is quite promising. It is further shown that Parzen window-based estimators for Renyi’s quadratic entropy and an integrated squared error (ISE) pdf divergence can also be expressed as functions of mean vectors in a Mercer kernel feature space. A new classification rule based on the ISE is proposed an studied theoretically. It is shown to be a hyperplane classifier in the kernel feature space, and a special case of the support vector machine (SVM). By introducing weighted Parzen window density estimators, an information theoretic interpretation of the SVM is provided. An application of independent component analysis (ICA) is studied. Image basis functions are created by presenting textured training data to the FastICA algorithm. These texture basis functions are shown to capture the properties of the texture, and are used as a filter bank for generating energy features for segmentation of textured images. The ICA filter bank yields similar or better results than the Gabor filter bank.